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SYSTEM AND METHOD FOR PROVIDING 
MULTI-PERSPECTIVE INSTANT REPLAY 

CROSS REFER ENCE TO RELATED APPLICATIONS 
This application claims priority to U.S. ProviiionaJ Patent Application 
No. 60035,529 entitled "SYSTEM AND METHOD FOR PROVIDING 
MULTI-PERSPECTIVE INSTANT REPLAY** filed September 26, 2000 
5 (ATTORNEY DOCKET NO. DFTVP014+), which is incorporated herein by 

reference for all purposes, and to U.S. Patent Application No. 09/630,646 
entitled -SYSTEM AND METHOD FOR INCORPORATING 
PREVIOUSLY BROADCAST CONTENT** filed August 2, 2000 
(ATTORNEY DOCKET NO. OPTVP013), which is mcoiparated herein by 
10 reference for all purposes. 

U.S. Provisional Patent Application No. 60/162,490 entitled 
"RECORDING OF PUSH CONTENT" filed October 29. 1999 (Client Docket 
No. OTV0033+), is incorporated herein by reference for all purposes). 

FIELD OF THE, CWEJTT1QT* 
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The present invention relates generally to interactive video delivery 
mediums such as inlerachve television, and more particularly, to a system and 
method for providing multi -persp ec ti ve instant replay of broadcast material 
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A broadcast service provider transmits audio-video streams to a 

viewer's television. Interactive television systems are capable of displaying 

text and graphic images in addition to typical audio-video programs. They 
5 can also provide a number of services, such as commerce via the television. 

and other interactive applications to viewers. The interactive television signal 

can include an mteractive portion consisting of application code, data, and 

signaling mftmnatiou, in addition to audio- video portions. The broadcast 

service provider can combine any or all of dus ^formation into a single signal 
10 or several signals for transmission to a receiver connected to the viewer's . 

television or the provider can include only a subset of the information, 

possibly with resource locators. Such resource locators can be used to indicate 

alternative sources of interactive and/or audio-video information For example, 

the resource locator could take the form of b world wide web rarvereal 
15 resource locator (URL). 

The television signal is generally compressed prior to transmission and 

transmitted through typical broadcast media such as cable television (CATV) 

lines or direct satellite transmission systems. Information referenced by 

resource locators may be obtained over different media, for example, through 
20 an ahvays-on return channel, such as a DOCSIS modem. 

A set top box connected to the television con tr o ls the interactive 

functionality of the television. The set top box receives the signal transmitted 
2 
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by the broadcast service provider, separates the interactive portion from the 
audio-video partkm, and deccOTTpresses the respective portions of the signal. 
Toe set top box uses interactive information to execute en appKcstion while 
the audio-video information is trensmittcd lo the television. Set top boxes 

5 typically include only i fimited amount of memory. While this memory is 

sufficient to execute interactive applications, it is typicalh/ not adequate to 
store the applications for an indefinite period of time. Further, the memory of 
the set top box is typically too small to accommodate a program which 
includes large amounts of audio or video data, application code, or other 

10 inferrnarion. Storage devices may be coupled to the set top box lo provide 

additional memory for (he storage of video and audio broadcast content 

lrrtcractive content such as application code or information relating to 
television programs is typically broadcast in & repeating format The pieces of 
information broadcast fn this manner form what is referred to as a M carouseT. 

I 5 Repeating transmission of objects in a carousel allows die reception of those 

objects by a receiver without reqiiihng ■ return path from the receivers to the 
server. If a receiver needs a particular piece of information, it can simply wait 
until the next time that piece of information is broadcast, and then extract the 
information from the broadcast stream. If the infonnation were not cyclically 

20 broadcast, the recover wouW have to transmit a request fo: the infonnatwn to 

the server, thus requiring a return palh. If a user is initially not interested in 
the carousel content, but later expresses an interest, the informahon can be 
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obtained the next time the carousel is broadcast Since broadcast networks 
have access only to a limited bandwidth, audio video content is not broadcast 
in carousels. There is also insufficient bandwidth and server resources to 
handle pulling of large amounts of data required for video and audio in real- 

S time to handle near simultaneous requests for broadcast of previously 

broadcast materia) from a vast number of television viewers. 

In a broadcast by a television network, such as a broadcast ofa 
sporting event, the content provider may generate multiple video feeds fiom 
various angles of the game, for example. The network may select one or more 

]0 feeds from the multiple video feeds and broadcast the selected video fted{i) to 

die viewing audience at any given point in time. That is, the network may 
simultaneously broadcast video tracks that present the same scene, except 
from a different perspective or send different audio tracks or subtitles if a 
movie is broadcast in different languages, for example. The viewer may use 

15 an interactive application that executes on their set top box to choose between 

different perspectives. When a viewer requests a change in perspective, the 
interactive application uses mcta-daia to determine which packets contain the 
chosen perspective. It starts delivering packets that contain the newly chosen 
p ersp e c tive. 

20 As previously described, a viewer cannot request previously broadcast 

audio or video material due to the limited bandwidth available on broadcast 
networks. Abo, data that accompanies interactive applications sometimes 

4 
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corresponds to radio cod video thai U currently being broadcast, to it changes 
rrcqucntry. In these cases, the values broadcast as pan of the carouse) often 
change and old values are no longer earned in (he carousel Thus, a viewer 
cannot replay t scene or a sporting event play from a different perspective 
5 unless the viewer has already recorded the video stream for the alternate 

perspective. 

SUMMARY OFTHT. INVENTION 

A method and system fin- providing multi-perspective instant replays 
are disclosed. A method tor processing broadcasts generally comprises 

1 0 receiving a broadcast of the program containing a plurality of perspectives of 

the program and presenting at least one of the plurality of perspectives to a 
viewer. The method may further include auimnancalry recording the plurality 
of perspectives in > storage device and playing alternate recorded perspectives 
for the viewer without interrupting the recording of the broadcast 

V5 The televirion program may comprise a plurality of related video 

streams, audio streams, executable code, and data. When appropriate, 
rpulripk perspectives may be displayed to a viewer simultarteoosJy by using 
picturr-withuv^picturt (PIP) window in a television screen. 

A system for recording a broadcast containing a plurality of 
20 perspectives of a program generally comprises a receiver operable to receive 

the broadcast, a storage device coupled to the receiver, and a processor 
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operable to present at least one of the plurality of perspectives to i viewer. 
The receiver may farther be operable to sutornarjcalty record the plurality of 
perspectivei in the storage device and play the recorded perspective to the 
viewer without interrupting the recording of the multiple perspectives. 

5 The receiver may be a set top box and the storage device may be 

ooutaiued within the set top box or coupled thereto. The storage device may 
comprise a magnetic disk, optical disk, or flash memory, for example. The 
receiver box may include one or more tuners. 

Other features, advantages, and embodiments of the invention will be 
10 apparent to those skilled rn the art from the following description, drawings, 

and claims. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a diagram illustrating the distribution of television programs 
1 5 and signaling information from a broadcast station to a receiving station. 

Fig 2 is a block diagram of a system of the present mvcntwnfor 
recording programs received from the b roadest! station of Fig. 1. 

Fig 3 is a block diagram illustrating the traiisfer of data to a storage 
device coupled to the set top box of Fig 2. 

20 Fig. 4 is a diagram flhmranrig three video streams and two audio 

streams simultaneously sent to a receiving station with one of the audio and 
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one of the video streams sent to a television. Those same streams are also sent 
to a storage device along with one of the other video streams. 

Fig. 5 is similar to the diagram of Fig. 4 except that the second video 
stream is now also displayed in a PIP window along with die first audio and 
S video itftrsros which are displayed in the main picture of the television. 

Fig. 6 is a diagram similar to the diagram of Fig. 5 except that the 
second video stream b now shown in the center of (he television screen with 
the first video stream shown in the PIP window. 

Fig. 6a is a diagram similar to the diagram of Fig. 6 except that the 
10 configuration shown docs not require or use a PJP. 

Ftg. 7 is a diagram similar to the diagram of Fig. 6 except that the live 
broadcast of the second video stream is replaced with a previously broadcast 
version of the same perspective. 

Fig. 7a is a diagram similar to the diagram of Fig. 7 except thai the' 
] S configuration shown does not require or use a PIP, and a recorded audio 

stream is played instead of a live audio stream as in Fig. 7. 

Fig. 6 is a diagram fflustreting a first video stream and audio stream 
displayed on a television and recorded along with a second audio stream. 

Fig. 9 is a diagram similar to the diagram shown in Fig. 8 except thai 
20 the first audio stream is replaced with the second audio stream. 
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Fig. 10 is a diagram similar to the diagram of Fig. 9 except thai the 
first video stream and second audio stream are replaced with earlier broadcast 
versions. 

Fig. 11 illustrates an example of files and data structures on a storage 
5 device. The text accompanying Fig. 1 1 describes how these data structures 

coold be used to facilitate the viewing of an instant replay from a different 
perspective. 

Fig. 12 is b flowchart of a method in accordance with the invention. 
Corresponding reference characters radicate corresponding pans 
10 throughout the several views of the drawings. 



Pmn.F.P DESCRIPTION Pf Tflfi n*VKNMION 

The following description is presented to enable one of ordinary skill 
in the art to make nod use the invention. Descriptions of specific 
15 embodiments and applications are provided only as examples and various 

modifications will be readily apparent to those drilled in the art The general 
principles described herein may be applied to other embodiments and 
applications without departing from the scope of the invention. Thus, the 
present invention is not to be limited to the embodiments shown, but is to be 
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accented ihc widest scope consistent with the principles and features described 
herein. U will be understood by one skilled in the pi that many e mb o dime nts 
are possible, such as the use of a computer system and display to perform the 
fim rtVw and features described herein. For purpose of clarity, the invention 

5 will be described in its application to a set top box used with a tdsviaton. and 

details relating to technical material that are known in the technical fields 
related to the invention have not been included. 

Referring now to the drawings, and first to Fig. 1, a diagram of a 
television broadcast and receiving system is shown and generally indicated at 

10 10. The system 1 0 include* a broadcast station 20 where audio-video and 

control information is assembled to the form of digital data and mapped into 
digital signals for satellite transmission to a receiving station. Control 
information such as conditianal acccas information and signaling information 
(such as a list of services available to user, event names, and schedule of 

1 5 event* (Hart time/dale and duration), and program specific information) may 

be added to video, audio, and interactive applications for use by the interactive 
television system. Control information can describe relationships between 
streams, such as which streams can be considered as carrying different 
perspectives of which other stream*. The control information is converted by 

20 the broadcast station to a format suitable for tnmamaaion over broadcast 

medium. The data may be formatted into packets, for example, which can be 
transmitted over a digital satellite network. The packets may be multiplexed 
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with other packet! fox transmission. The signal is typically compressed prior to 
transmission and may be transmitted through broadcast channels such as cable 
television lines or direct satellite tnmxmission systems 22 (as shown m Fig. 1). 
The Internet, telephone lines, cellular networks, fiber optics, or other terrestrial 
transmission media may also be used in place of the cable or satellite system 
for transmitting broadcasts. The broadcaster may embed service information 
in the broadcast transport stream, and the service information may list each or 
the elementary s tream identifiers and associate with each identifier an 
encoding that describes the type of the associated stream (eg, whether it 
contains video or audio) and a textual description of the stream that can be 
understood and used by the user to choose between different perspectives, as 



The receiving station includes a set lop box 16 connected to a storage 
device IB, and a television 20 which is used to present programs to a viewer. 

15 The set top bent 1 6 is operable to decompress the digital data and display 

programs to a viewer. The decompressed video signals may be converted into 
analog signals such as MTSC (National Television Standards Committee) 
format signals for television display. Signals sent to the set top box 16 ere 
filtered and of those that meet the filtering requirements, some are used by the 

20 processor 30 immediately and others can be placed in local storage such as 

RAM. Examples of requirements that would need to be filtered for include a 
particular value in the location reserved for an elementary stream identifier or 
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■0 originating network identifier. The set top boa 16 may be used to overlay 
or combine different signals lo form the desired display on the viewer's 
television 20. 

As farther described below, the set top box 16 unconfigured to record 

5 one or more video and/or audio streams simultaneously to allow s viewer to 

replay a scene which has recently been viewed or heard by a viewer, except 
from a different perspective. Broadcast station 12 simultaneously broadcasts 
multiple perspectives fig use by viewers that have set top boxes 16 which 
execute interactive tekvisiOD applications. For example, multiple cameras 

10 may be used to record & sporting event snd the station may broadcast from the 

multiple cameras at the same time to aOow the viewer lo choose between 
different camera views using an interactive application that executes on their 
set top box 16. A broadcaster may also send multiple perspectives of andio 
tracks in different languages, for example. The multiple video and audio 

1 5 perspectives are only examples of types of perspectives or which a plurality 

may be contained in a broadcast. Other examples include multiple teletext 
streams, perhaps in different languages; multiple executable*, perhaps each 
meant for a different skill level; or multiple data streams. The present 
invention allows a viewer to replay the same s ce ne from a different 

20 perspective, while ensuring that the viewer will still be able to view, cither 
simultaneously or at a later time, the portion of (he program being broadcast 
simultaneously with their viewing of the replay. The viewer may request a 
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replay of any combination of audio, video, crccutabies, and data, from either 
the Bene or diffarent perspectives as the perspectives previously played. 

U is to be underatood that the term "program" at used herein refer* to 
any broadcast material including television shows, spoiling events, news 

5 programs, movies, or any other type of broadcast material, or a segment of the 

material The material may include only audio, video, data, or any 
combination thereof. The program may be only a portion of a television show 
or broadcast (eg., without cornrnercials or missing a portion of the beginning 
or end) or may be more than one ihow, or include commerciab for example 

10 Furthermore, it ii to be understood that the term "viewing" as used herein is 
defined such that viewing of a program begins as soon as a tuner begins 
filtering data corresponding to a program. If a viewer has tuned to a particular 
frequency prior to (he broadcast of a program, the beginning of the viewing 
preferably corresponds to the beginning of the program. The viewing 

15 preferably ends when the prog r am is complete or when the tuner is no longer 

filtering the frequency corresponding to the program. Thus, the recording of a 
program coincides with the •Viewing" of a program and the program b only 
recorded when a tuner b tuned to the station broadcasting the program. In the 
event that the television display is turned off after a viewer has started 

20 recording the program, as long as the tuner is tuned into me station 

broadcasting the pr og ra m and a recording of the information broadcast on the 
same frequencies as those used at the start of the viewing is being made, the 

12 
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viewing is wid to conJiimc The «udk)-vklco signals and program control 
ngn&h received by the set top box 1 6 correspond to lehwsiOT programs and 
menu selections thai the viewer may access through a user interface. The 
viewer may control the sot top box 16 through an infrared remote control unit, 

5 a control panel on the set top box, or a menu displayed on the television 

terpen, for example. 

It is to be understood (hat the system 10 described above and shown in 
Fig. 1 is only one example of a system used to convey signals to the television 
20. The broadcast network system may be different than described herein 

10 without departing from the scope of the invention. 

The set top box 16 may be used with a receiver or integrated decoder 
receiver that is capable of decoding video, audio, and data, such as a digital set 
top box for use with B satellite receiver or satellite integrated decoder receiver 
that is capable of decoding MPEG video, audio, and data. The set top box 16 

15 may be configured, fin- example, to receive digital video channels which 

support broadband communications using Quadrate Amplitude Modulation 
(QAM) and control channels for two-way signaling and messaging. The 
digital QAM chance la carry compressed and encoded multiprogram MPEG 
(Motion Picture Expert Group) transport streams. A transport system extracts 

20 the desired program from the transport stream and separates the audio, video, 

and data components, which are routed to devices that process the streams, 
such as one or more audio decoders, one or more video decoders, and 
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optionally to RAM (or other fonn of memory) or b bard drive ltistobe 

understood to the set top box 16 and storage device 18 may be analog, 

digital, or both analog and digital. 

As shown in Figs. 1 and 2, me storage device 18 is coupled to the set 
5 top box 16. The storage device 18 bused to provide sufficient storage to 

record program* that win not fit in the limited amount of main memory (eg.. 

RAM) typically available in set top boxes. The storage device 18 may 

c om p rise any suitable storage device, such as a hard disk drive, a recordable 

DVD drive, magnetic tape, optical disk, magneto-optical disk, flash memory, 
10 or solid stale memory, for example. The storage device 18 may be internal to 

the set top box 16 connected externally (eg., through an IEEE 1394-1995 

connection) with either a pcrmancn! cormectkm of a removable connection. 

More than one storage device 18 may be attached to the set lop box 16. The 

set top box 16 andVor storage device 18 may also be included m one package 
1 5 with the television set 20. 

Fig. 2 illustrates one embodiment of a system of the present invention 

used to record programs received from the broadcast station 12. The set top 

box 1 6 generally includes a control unit (e.g., microprocessor), main memory 

(eg., RAM), and other eompoDcnla which are necessary to select and decode 
20 the received interactive television signal As shown in Fig. 2, the »et lop box 

16 includes a frost end 26 operable to receive audio, video, and other data 

from the broadcast station 12. The broadcast source is fed into the set top box 
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16 el (he finot end 26, which comprises in analog to digital (A/D) converter 
and tusa/donodulaton (not shown). The front end 26 filters out a particular 
band of freq u encies, demodulates it and converts h to a digital formal. The 
digitized otrtpmistben sent to a transport stage 28. The transport stags 28 

5 fhrtiw processes the data, sending a rwrtionoflhedautoan audio-visnal 

(A V) stage 34 tor display and another portion to the control processor 30, and 
filtering out the rest of the data. 

Control information may also be recorded as broadcast along with the 
audio-video data or may be first manipulated by software within the set top 

10 box 16. For example, broadcast CA (conditional access) mformation may be 

used to decrypt broadcast video. The original broadcast streams, or 
modifications of these streams may be optionally re-encrypted using a set top 
box key or algorithm prior to recording. The encrypted video may also be 
stored as received along with the broadcast CA informstion. Also, clock 

IS information may be translated to a virtual time system prior to recanting. An 

MPEG-2 elementary stream may be de-multiplexed from an MPEO-2 
transport stream, men encapsulated as a program stream and recorded. 

Fig. 3 iltuarates the transfer of data from the transport ctegc 28 to the 
storage device 18. The storage device 18 typically contains a plurality of 

20 programs which have been recorded by a viewer. The recordings of each 

perspective are associated with identifying information that may have been 
copied or modified tram the original signaling information. This identifying 

15 
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. information may contain bookkeeping mfonmlion similar to thai typically 

sored in audio/video file systems or hierarchical computer file system*. The 

identifying information may have various formats and content, ts long as ft 

provides sufEctcnl information to allow the viewer, possibly interacting with 
S the system, to uniquely retrieve a panicular recorded perspective. The 

programs may be identified with art ID number and a start time and end time. 

As described below, the storage may be defragmented periodically so that the 

programs are stored in a contiguous manner. Direct memory access (DMA) is 

preferably used to send data from the transport stage 28 to the storage device 
10 18. The data that is seal to the control pro cess or 30 may include mcta-data 

which describes the content of the audio-video data streams and may also 

include application programs and correspcDdmg data lhai can be executed oo 

the control processor in order to provide interactive television. 

A copy of data sent from the transport stage 28 to the A V stage 34 is 
IS sent to the storage device 18 at the begrnnmgoftl»e viewmg. TbeC^mtbe 

control processor 30 configures a DMA controller to ensure thai Ok data is 

written to a buffer Out is allocated in Ox storage device 1 8. The Dumber of 

minutes of viewing data to be recorded in the buffer is preferably selected by 

the viewer; however, the set top box may 16 be preset with a default value 
20 such u fifteen minutes. The control rmxessor's CPU calculates the size of the 

buffer to allocate based upon the number of minutes and Pic maximum speed 

at which bits in the transport stream that the viewer is watching wiU be sent 
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This maximum speed may be orrtained from meia-data sent with the audio- 
video stream. When the cod of the buffer is reached, the CPU m the control 
processor is interrupted, at which tune it will re-configure the DMA controller 
to start writing at the beginning of the buffer. This design is known as a 

5 circular buffer. 

The buffer is preferably circular to allow contiguous recording and 
writing over of previously recorded content When the viewer changes (he 
channel or a TV event (eg., television program ends) occurs, the control 
processor's CPU will be mterruptcd. At Oris time, the CPU may allocate a 

10 new buffer or mark die beginning of the new event in the original buffer. The 

automatic recording of a pro g ram and aD related video, audio, and data 
streams in a storage device at the start of the program without any action by 
the viewer, allows the viewer to replay a portion ofihe program Gum a 
different perspective* 

1 5 As previously described, the control processor 30 records the multi- 

perspective streams at a start of (he program to store the perspectives in 
storage device 18. The perspectives will cc*rtinuc to be recorded and stored 
within the storage device 18 for a pre-determined period of time (eg, 15 
minutes). If a viewer decides to record the entire viewing after the start of the 

20 program, he will select a record option and the processor 30 will allocate space 

within the storage device 18. All perspectives will be recorded along with the 
program that is being viewed. See cg^ U.S. Patent Application Serial No. 
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09/630,646, entitled "System and Method for Tncwporating Previously 
Broadcast Contcnr and filed August 2, 2000 (Attorney Docket No. 
OFTVP013), which is incorporated herein by icfc im t t in hs entirety. 

The joining of the first and second recorded portions of any given 

5 perspective in a common storage area may be implemented either physically 
or virtually. A physical implementation may include copying the first 
TeconJcd portion to a location where the second portion has been recorded. A 
virtual implementation may include the modification of a data structure stored 
in a storage device. In cither case, a viewer watting a replay of any 

10 perspective should not be able to detect that the two parts of the perspective 

were originally stored separately. Thus, the portions of the perspective may be 
physically contiguous or the portions of the perspective may be stored 
separately in a non-contiguous format as long as the entire recorded program 
can be played back in a continuous manner (i.e., viewer does nol notice a 

15 transition between the playback of the first and second portions of the 

perspective). . 

It is to be understood that the recording or the entire program, 
including the plurality of perspectives, fn the storage device 18 may occur 
without any action by the viewer. For example, if the viewer rewinds (or 
20 performs a similar action an different types of storage media) a portion of one 

of the recorded perspectives to replay a scene, the entire program along with 

18 
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ill of its multiple perspectives may be recorded m the storage device, since the 
viewer has shown interest in the program. 

The control information that is broadcast with the program preferably 
mdicalea which streams are related to the viewed streams. The aet top box 16, 

5 by filtering on the appropriate identifiers in the broadcast MPEG-2 (or DSS or 

other aicodmg) packets can locate ail related ekmcnary streams. 11 sends the 
streams that the viewer a watching to the tdevixion set 20 end records in the 
storage device 1 8 the context of these streams, along with the other related 
streams, irjcfoding related video, audio, executable*, and data. Meta-data that 

10 indicates the maximum bit rate for the streams may accompany the elementary 

or transport st ream s. The format of the recorded streams may depend upon the 
hardware support. For example, special purpose hardware inside the set top 
box 16 may support re-mitfUplexing of streams or concurrent reads and writes 
to the storage device 18, as is well known by those skilled in the art 

15 Broadcast data such as audio and video data, application code, control 

signals and other types of information may be sent as data objects. If Ihe 
program is to be consumed (Le., presented to the viewer) the broadcast data 
must be parsed to extract data objects from the stream When the necessary 
data objects have been extracted, the program is played. For example, any 

20 applications that need to be executed are launched and any audio or video data 

thai needs to be presented to the viewer is played. If Ihe program is stored, the 
data objects are extracted in the same nianner, bu! they arc stored instead of 
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being immediately used to present the program. The recorded program n 
played back using the stored data object*. The data objects may include Hive" 
data which became* obsolete if not consumed immediately. If this data is 
stored and used when the program is played back, the program will in at least 
pan, be obsolete. Thus, while most of the data objects may be stored as files, 
bvc data objects may be stored as re fe rences in the program. When the 
program is played back, new live data corresponding to the reference may be 
obtained and used in place of the data which was live al the time the program 
was recorded. Thus, only temporally correct data is used by the interactive 
application when it executes at a later tunc. (See eg., US. Provisional Patent 
Application No. 60/162,490 entitled "RECORDING OF PUSH CONTENT" 
filed October 29, 1999 (Client Docket No. OTV0033+), which is mcorporared 
herein by reference tor all purposes). 

Figs. 4-10 show the set top box 16 receiving three video and two audio 
streams fiom the broadcast station 12. The signals are received from the 
broadcast station 12 at the tuner in front end 26 and related streams arc sent to 
demultiplexer and processor 100. Video streams VI, V2 and audio stream Al 
are all related (eg., video streams are different camera views of a sporting 
event and Al is the sound track for the announcer) and can be provided in a 
single transport stream. IT all the related streams are provided in one transport 
stream only one toner 50 is required. The set top box 16 may include multiple 
tuners SO for recording and displaying related streams broadcast in separate 
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transport stream*. Related sTreann are preferably broadcast on a small number 
of frequencies so lhal a large number of tuners will not be required within or 
attached to the set top box 16. For example, a large number (e.g., five) of 
video streams along with multiple audio streams, executable programs, data, 

5 and control information may be multiplexed together on a single frequency. 

Figs. 4-7 illustrate • ease where a viewer requests a replay from a 
(afferent perspective using a p«un>witon-pictare (PIP) mode. If a viewer 
wants to see the replay from a different perspective, it can be viewed in a PIP 
mode without requiring multiple tuners inlheaettopbox!6oTthe television 

10 2a The additional tuner is not required since one of the video or audio 

streams that bad been previously recorded is coming from the storage device 
18. All streams shown are preferably multiplexed on the same frequency. The 
video or audio can be delivered directly to the AV stage 34 which is contained 
in 100 which itself is inside the set top box 16, and may be multiplexed with a 

IS transport stream mat is being delivered via the tuner 50. Note that 1 00 

represents three cmnponents: (i) a demultiplexer, (ii) a processor that directs 
portions of Cttt broadcast information to other components; and (iii) an AV 
stage that modulates when accessary (U when the television is analog). 
Alternatively, the viewer can choose to view only the replay while the set top 

20 box 16 buffers, on the storage device 18, the live broadcast for later delivery, 

as described below with respect lo Figs. 8-10. 
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In Fig. 4, the broadcast Italian 12 U sending video streams VI and V2 
containing two different perspectives and one audio stream Al. The two 
video streams may be two different camera positions al a baseball game, for 
example. The viewer i$ currently watching video stream VI and littering to 

5 audio Cream Al. The first and second video streams VI and V2 and the audio 

stream Al are automatically recorded. Thus, the previously broadcast 
information is available if a viewer warts to replay, for example, the hat play 
of the game. In particular, with this invention, the viewer can replay thU 
m format km from any of the previously broadcast perspectives. The viewer 

10 may place the scl top box into a PIP mode so that the viewer can see a first 
perspective (video stream VI) displayed in a large central area in the 
television screen and a second perspective (video stream V2) displayed in a 
small picture window in Ihe top right hand corner (or some other area) of the 
television screen (Fig. 5). After an important play in the game (eg., doable 

15 play in a baseball game), the viewer may want to see a replay, this time from a 

perspective different from the or* shown in VI. At this time, the viewer may 
optionally swilch the windows into which the video streams V i and V2 are 
displayed, bs shown m Fig. 6. Video stream VI is now sent to the PIP 
window and video stream V2 is sent to the centra] viewing window. Then 

20 the viewer would give a command (i.e. press a button on the remote control) to 

re-wind the video in the main window while permitting the PIP window to 
continue displaying the "live" VI in the PIP window. 
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As ihown in Fig. 7, the recorded video stream V2\ which is from the 
same perspective as V2, but which was broadcast and recorded earlier, is sent 
from the storage device 18 to the demultiplexer in 100 which sends the 
previously recorded stream V2' along with the current video stream VI to the 
television for display. The viewer may rewind or search through the recording 
until the beginning of the recording is reached. The viewer may also rewind 
and display the first video stream VI. Meanwhile the broadcast of the 
remainder of the program may be sent to the storage device IS since the 
viewer has shown an interest in the recording. This may be automatic (Le-, 
program streams are sent to storage device 18 upon a viewer's request for a 
replay) or may only occur upon receiving a request from the viewer to record 
the entire program. 

Alternatively, a viewer may prefer not to be distracted by the live 
broadcast which is shown as being displayed in the PIP in Figure 7. 
Therefore, the viewer may simply first switch penpectives from VI to V2 as 
shown in Figure 6a. After that, the viewer rosy "rewind" to an earlier evcnJ to 
see a previous scene from the perspective carried in video stream V2. This 
case is shown in Figure 7a where a copy of the live video stream VI is only 
sent to the storage device, along with the live video stream V*2 and live audio 
stream Al. The recorded streams V2* and Al ' are the only ones sent, possibly 
after modulsoon, to the television. The scenario presented in figures 6a and 
7a could also be a scenario used by the viewer to switch between a live video 
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perspective and a different, recorded, video petipective, when there is no PIP 
functianality associated with the viewer 1 * television. 

Figi. 8-1 0 illustrate a case where a program is broadcast with 
different perspective audio streams. For example, a viewer may be watching 
S an Italian movie that is broadcast with an Italian audio stream A 1 and an 

English andio stream A2. As shown in Ftg, a, video stream VI end audio 
stream A I are presented to the viewer and recorded in the storage device 18 
while audio stream A2 is alsn recorded in the storage device 18 but not 
presented to the viewer. The viewer is initially listening to the Italian 
10 broadcast (audio stream A 1 >; however, during Dart of the movit^ the viewer 

does not understand the Italian, so he selects a "switch to English** option from 
a menu and the viewer now hem the English broadcast (audio stream A2) 
(Fig. 9). If the viewer wants to hear the soundtrack that accompanied the 
previous scene in English, he may rewind the tape or the video stream VI and 
15 audio stream A2 and watch the scene over again in English (Fig. 10). The 

video and audio streams VI, Al, and A2 will oomwoe to be recorded so thai 
the viewer can see the rest of the movie in a deferred mode, without missing 
the potion of the movie that was broadcast while the viewer was rewinding 
and replaying the previous scene. 
20 Fig. 1 1 shows an example of a meta-data file that can be stored along 

with each recorded perspective. This invention does not require the format 
shown in this figure, but the format is only used as an example of how meta- 
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data can facilitate the playing of an instant replay from a different perspective. 

Each record of the meta-data file shown contains, among other possible fields, 

a time and an offset In this example, a program clock reference is frequently, 

though not periodically, broadcast along with the video. When tome of these 
5 clock reference values are received by the set top box, theii value, akmg with 

the offset into the recording of the most recent I-fraroe (one of 3 types of 

MPEG-2 frame encodings that can be used for video), can be recorded as 

meta-data. Again, this is only an cxunple; an actual implementation may 

make use of P- and B- frames (the other types of MPEG-2 encodings, both of 
10 which are typically more compressed than an 1-frime). The offset is rn terms 

of bytes measured from the beginning of the file containing the recording of 

the perspective. 

In this example, the viewer baa been waiching a live broadcast that 

contains video perspective V,. As the viewer watches, that video perspective, 
15 V t is being recorded to a file, Abo, other video perspectives, mdudiiig video 

perspective V,, axe being recorded to a different file because they represent a 

different view 0 r the same ^formation. Of course, V 2 could be recorded in the 

same file as long as other information distinguishing V t from V 2 is recorded 

Kjmcwhere, The viewer has just seen something interesting on the screen and 
20 enters die appropriate comnumds to cause Vi to be re-wound to the beginnirrg 

of the interesting scene. The viewer stops V, when the MPEG-2 I-Frame,,, is 

being used to display the contents of the screen. (Again, this is only an 
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example. P-tndB- frames could also be recorded in the file containing the I- 

frame* from V,, and could be used in locating a scene, but they are not used in 

Out example. Abo, MPEG-2 is only used as an example; other formats of 

media and/or data can equally well be used) The viewer then issues a 
5 command that lelli the set top box to start playing forward, but from V 2 rather 

(ban from V|. The set-top box must determine which I-frame of V 2 it should < 

first cause to be displayed. A simple solution, choosing (be 1- frame nearest to 

the same offset as I-frame^ in the file that cemtams V 2 would only work 

correctly if both perspectives were sent at the same constant rate, although 
10 such an aixproximafion may be useful if the perspective* were leal at 

approximately the same non-constant rate. A better solution for either 

variable-rate streams or streams with different constant rates is now presented. 

This solution uses a linear interpolation, although other well-known classical 

interpolation methods that arc readily available in the open literature may 
I 5 provide a better approximation under some circumstances. 

First the actual time corresponding to the originally intended playing 

time of I-frame^ is approximated The oflset into the fite containing V, 

where I-frame),, is located, d u is used for this approximation. In order to 

approximate this time, t, two ccmsccutive time values, d^and di>+i, are 
20 searched for in the mcta-oata file, such that di, £ d u < dj>.,. (As a practitioner 

of the art would know, a binary search would likely find these two consecutive 

dements the most quickry if the records are fixed length and the elements are 
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stored in ronsccutive onto as shown. A different search would be optima) if a 
different storage fonnit is used. Again, these tre well-known technique* that 
are extensively documented in the computer science literature.) Oncctbey 
ore located, both t,^ and t 1>+ , will also be known. These values ire then used 
to approximate L This example uses the linear interpolation formula: 

t - ( Oi>i - li j ) (d,j- d M ) / (di>*, - d,j ) ) + 1 W 
After an approximation for t has been found, (he location of the I- 
frame in the recording of perspective V 2 that is nearest to that time needs to be 
found. The firel step here is to locate and ti», snch that ti>3t<tu*i. 
Again, the search that performs the beat in any given ease is dependent upon 
the formal of the file and is a well-studied problem. Having these values 
allows for an approximation of <h,L Once again, this example uses linear 



<hW(d*r*i-fcj)(t-l*)/(t2. r > -ty)) + dy 

Mow mat ar. approximation foT du is known, the 1-fraroc mat is nearest 
to being du bytes from the beginning of the file com aining the recording of V, 
is used as the starting frame for playing back the recording for the viewer. 



Fig. 12 shows a process flow in accordance with the embodiment 
described hernia For the sake of clarity, the process has been illustrated with 
a specific flow, but it should be understood thai other sequences are possible 
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and that some may be performed in parallel, without departing from the spirit 



multiple perfective* of a program. The system presents one of the 
perspectives to the viewer, step 2 10, and stores all of the perspectives in s 
storage device, step 220. m the embodiment disclosed, the system stores all of 
the perspectives, hut may be configured to selectively store perspectives based 
on criteria provided by the viewer (such as an indication of which perspectives 
the viewer is interested in). The perspectives air stored in a circular buffer, 
step 260. Another perspective is presented to the viewer, step 230, and the 
presentation of this perspective and the first perspective includes preparation 
of an audio/video signal for the television, step 250. The presentation of the 
other perspective in step 230 may involve searching the stored perspectives, 
step 240, and the perspective presented may be one of the stored perspectives. 

A method and system for processing broadcasts have been disclosed. 
Software written according to the present invention may be stored in some 
form of computer -readable medium, such as memory or CD-ROM, or 
transmitted over a network, and executed by a processor. Additionally, where 
methods have been disclosed, various sequences of steps may be possible, end 
it may be possible to perform such steps simultaneously, without departing 
from the scope of the invention. 

Although the present invention has been described in accordance with 
the embodiments shown, one of ordinary skill in the art will readily recognize 



of the invention. In step 200, the system receives a 



including 
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thu there could be variations made to ihe cmbodnneots without departing 
from the scope of the present invention. Acronimgly, it is intended that aU 
maner contained in the above description and shown in the accompanying 
drawings shall be interpreted as UlnstiaiWe and not m a hmitixig tense. 

5 
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CLAIMS 

WHAT IS CLAIMED IS: 

1. A method for processing broadcasts, w»n« Jing: 

receiving a broadcast of a program, the broadcast containing a plurality 
of perspectives of the program; 

presenting a first of the plurality of perspectives to a viewer, 
storing at least one of the plurality of perspectives; and 
presenting at least one of the plurality of perspectives to the viewer. 

2. Tbe method as recited in claim 1 , wherein presenting Ihe first 
perspective and storing are performed simultaneously. 

3. The method as recited in claim 1, wherein storing Ihe at least one 
of the plurality of perspectives is performed automatically. 
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4. The method wrecitcdm claim 1, wherein presenting ai loBtcnc of 
(he plurality of perspectives includes presenting el least one of ihe stored 
perspectives. 

5. The method &s recited id claim 4, wherein presenting at least one of 
the plurality of perspectives end storing ere performed simultaneously. 

6. The method as recited in claim 5, wherein storing at least one of 
the plurality of perspectives includes storing aD of the perspectives. 

7. The method as recited in claim I, wherein receiving the broadcast 
includes receiving a plurality of related video streams, each stream mchding 
one of the perspectives. 

8. The method as recited m claim 1, wherein receiving the broadcast 
includes receiving a plurality of related audio streams, each stream including 
one of the perspectives. 

9. The method as recited in claim 1, wherein presenting the first 
perspective includes presenting the first perspective in one window of a 
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display and presenting at least one or the plurality of perspectives includes 
presenting a srcrmri perspective from the plurality of stored perspectives in a 
different window of the display. 

10. The method bs recited m claim 1, wherein storing at least one of 
the plurality of perspectives includes storing the perspectives in at least one 
circular buffer. 

11. The method as recited in claim I, wherein presenting at least one of 
the plurality of perspectives includes preparing a video signal for display on a 



12. The method as recited in claim 1. wherein presenting at least one 
of the plurality perspectives includes the step of preparing an audio signal 
for a speaker. 



13. The method as recited in claim 4, wherein presenting Hi least one 
of the plurality of perspectives includes sending at least one of the stored 
perspectives from a storage device through a demultiplexer to a display. 

2G 
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14. The method as recited in claim 4, wherein presenting el teas: one 
of the plurality of perspectives includes searching one of the stared 



5 1 5. A method for playing a mutti -perspective program comprising: 

receiving a broadcast of (he program, the broadcast including a 
plurality of perspectives of the piugiun; 

recording at feast one or (be plurality of perspectives in a storage 

10 sendinga first of trie phinaliry of perspectrves to i display to phyet 

least a portion of the program; and 

replaying the portion of the program from a different perspective by 
sending a second of the plurality of perspectives from the storage device to the 
display. 

15 

16. Tr* method as recited in claim 15, wherem we ^perspective is 
a recorded perspective. 

17. The method as recited in claim 15. further comprising sending the 
20 second perspective to one window in the display to play the portion of the 
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program from the secood perspective concurrently with sendmg the first 
perspective to & differed window in the display. 

18. The method as recited in claim 1 5, wherein receiving the broadcast 
includes receiving at least ooe video stream, each video stream being 
associated with one of the perspectives. 

19. The method as recited in claim 1 5, wherein receiving the broadcast 
includes receiving at least one audio stream, each audio stream being 
associated with one of the perspectives. 

20. A system for recording a broadcast including a plurality of 
perspectives of a program, comprising^ 

a receiver operable to receive the broadcast; 

a storage device coupled to the receiver, 

a processor operable to present at least one of the plurality of 
perspectives to a viewer, record at least one of the plurality of perspectives in 
the storage device, and present at feast one of the recorded perspectives to the 
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21. The system as recited in dsim 20, further configured to record the 
perspectives auto m a t ically. 

22. The lyacm as recited in claim 20, further configured to present the 
5 recorded perspectives to the viewer without interrupting the recording of the 

23. The system as recited in claim 20, wherein the receiver is » wt top 

box. 

10 

24. The system as recited in claim 22, wherein the storage device is 
contained within the set top box. 

23. The rystem is recited in claim 22, wherein the storage device U 
IS removabry coupled to the set top box. 



26. The system as recited in claim 20, wherein the storage device 
comprises a magnetic d rtk - 
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27. The system as recited in claim 20, wherein the storage device 
comprises an optical disk. 

28. The system is reciled m claim 20, wherein the storage device 
5 comprises flash memory. 

29. The system as recited m claim 20, wherein the receiver comprises 

10 30. The system as recited in claim 29, wherein the receiver comprises 

a demultiplexer and a processor. 

31. A system for presenting broadcasts, comprising: 
a receiver configured to receive a broadcast including a plurality of 
1 5 perspectives of a program; 

a storage device for storing at least one of the plurality of perspectives; 

and 

a processor configured to present s first perspective to a viewer while 
presenting at least a second, stored perspective to the viewer. 
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32. The system as redtedm claim 31, wherem the first perspective is a 
stored perspective. 

33. The system as recited m claim 31, wherein the storage device is 
5 configured to store the at least one perspective automatically. 

34. The system as recited in claim 3 1 , further configured to store the at 
least one perspective simultaneously with presenting the first perspective. 

10 35. The system as recited in claim 31, further configured to store the at 

least one perspective simultaneously with presenting the second perspective. 

36. The system as recited in claim 31, further comprising a display 
coupled to the processor for presenting si least one perspective. 

15 

37. The system as recited in claim 36, wherein the processor is 
configured to present the tint perspective in a first window on the display and 
the second perspective in a second window on the display. 
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38. The system as recited in claim 37, wherein one of the first and 
second windows is nested inside the other of the first and second windows. 

39. The system as recited in claim 31, wherein the receiver is 
configured to receive a plurality of video streams associated with the plurality 
of perspectives. 

40. The system as recited in claim 39, wherein each video stream 
includes one of the perspectives. 

41 . The system as recited in claim 31, wherein fhe receiver is 
configured to receive a plurality of audio streams associated with the plurality 
of pospectj vet . 

42. The system as recited in claim 4 1 , wherein each audio stream 
includes one of the perspectives. 

43. The system as recited in claim 31, wherein the storage device 
includes at least one circular buffer for storing at least one of the plurality of 
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44 . The system as red tod in claim 3 1 , wherein the processor is 
configured to search al feast one of the stored perspectives. 



45. A 



for processing broadcasts, comprising 




receiving a broadcast ofa program, the broadcast containing a phiraKty 
of p e rspec tives of the program; 

presenting a first of the plurality of perspectives to a viewer, 
storing at least one or the proraiity of perspectives; and 
presenting ai least one of the plurality of perspectives to the viewer. 



46. The ccanptrterrjiograro product as recited in claim 45, wherein the 



47. The computer program product as recited in claim 45, wherein 
storing at least one of the plurality of perspectives is performed automatically. 
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48. The cfffnp'rtcr program product as recited in claim 45, wherein 
presenting at least one of the plurality of penpectives includes presenting it 
least one of the stored perspectives. 

49. Tbc computer program product as recited in claim 48, wherein 
presenting at least one of the plurality of peispectrves mid rtorunj m least one 
of the plurality of perspectives are performed simultaneously. 

50. The computer program product as recited in claim 49, wherein 
storing at least one of the plurality of perspectives includes staring all or the 
perspectives. 

5 1 . The computer program product ts recited in claim 45, wherein 
receiving the broadcast includes receiving a plurality of related video streams, 
each stream including one of the perspectives. 

52. The cocnpuler program product as recited in claim 45, wherein 
receiving the broadcast includes receiving a plurality of related audio streams, 
each stream including one of the perspectives. 
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53. The compiler program product as recited in claim 45, wherein 
presentin g the first perspective includes presenting the firsl perspective in one 
window of a display and presenting tl last one of the plurality of perspectives 
includes presenting b second perspective from the plurality of stored 
perspectives ins different window of the display, 

54. The computer program product as recited in claim 45, wherein 
storing at least one of the plurality of perspectives inchides storing the 
perspectives in at least one circular buffer. 
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